Inserting imagery from a real environment into a virtual environment

ABSTRACT

The present disclosure relates to techniques for inserting imagery from a real environment into a virtual environment. While presenting (e.g., displaying) the virtual environment at an electronic device, a proximity of the electronic device to a physical object located in a real environment is detected. In response to detecting that the proximity of the electronic device to the physical object is less than a first threshold distance, imagery of the physical object is isolated from other imagery of the real environment. The isolated imagery of the physical object is inserted into the virtual environment at a location corresponding to the location of the physical object in the real environment. The imagery of the physical object has a first visibility value associated with the proximity of the electronic device to the physical object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Stage patent application ofPCT/US2019/050008, entitled “INSERTING IMAGERY FROM A REAL ENVIRONMENTINTO A VIRTUAL ENVIRONMENT,” filed on Sep. 6, 2019, which claims thebenefit of U.S. Provisional Patent Application No. 62/728,610, entitled“TRANSITIONING BETWEEN IMAGERY AND SOUNDS OF A VIRTUAL ENVIRONMENT AND AREAL ENVIRONMENT,” filed on Sep. 7, 2018, U.S. Provisional PatentApplication No. 62/729,154, entitled “INSERTING IMAGERY FROM A REALENVIRONMENT INTO A VIRTUAL ENVIRONMENT,” filed on Sep. 10, 2018, andU.S. Provisional Patent Application No. 62/892,870, entitled“TRANSITIONING BETWEEN IMAGERY AND SOUNDS OF A VIRTUAL ENVIRONMENT AND AREAL ENVIRONMENT,” filed on Aug. 28, 2019, which are hereby incorporatedby reference in their entireties.

FIELD

This application relates generally to virtual environments, and morespecifically to techniques for inserting imagery from a real environmentinto a virtual environment.

BACKGROUND

Computers can completely project or partially superimposecomputer-generated images on a user's view to provide a virtualenvironment that can be experienced by the user. A virtual environmentcan be based on different types of realities. An electronic deviceoptionally detects the user's real movements and projects and simulatesthose movements within a series of visual images or video of the virtualenvironment. Through these movements projected or simulated within thevirtual environment, the user can appear to move to different locationswithin the virtual environment.

BRIEF SUMMARY

The present disclosure describes techniques for inserting imagery from areal environment into a virtual environment. The imagery from the realenvironment is inserted in response to an electronic device detectingthat a user (and/or the device) is approaching a physical object in thereal environment. Imagery of the physical object is then isolated andinserted into the virtual environment. These techniques, as describedherein, provide the user with an enhanced degree of safety by, in someexemplary embodiments, providing imagery of the physical object beforethe user comes into contact with the physical object.

In accordance with some embodiments, a method is described. The methodincludes: at an electronic device having one or more displays:presenting, using the one or more displays, a virtual environment;detecting a proximity of the electronic device to a physical objectlocated in a real environment; in response to detecting that theproximity of the electronic device to the physical object is less than afirst threshold distance: isolating imagery of the physical object fromother imagery of the real environment; presenting the virtualenvironment with the isolated imagery of the physical object insertedinto the virtual environment at a location corresponding to the locationof the physical object in the real environment, the imagery of thephysical object having a first visibility value associated with theproximity of the electronic device to the physical object; and inresponse to detecting that the proximity of the electronic device to thephysical object is greater than the first threshold distance: presentingthe virtual environment without the isolated imagery of the physicalobject.

In some embodiments, the method further includes: in response todetecting that the proximity of the electronic device to the physicalobject is less than the first threshold distance: displaying the virtualenvironment at a second visibility value associated with the proximityof the electronic device to the physical object. In some embodiments,the method further includes: in response to detecting that the proximityof the electronic device to the physical object is less than a secondthreshold distance: modifying the first visibility value of the imageryof the physical object. In some embodiments, the method furtherincludes: in response to detecting that the proximity of the electronicdevice to the physical object is less than a third threshold distance:ceasing to present the virtual environment; and providing a view of thereal environment.

In some embodiments, presenting the virtual environment with theisolated imagery of the physical object includes compositing imagery ofthe virtual environment with the isolated imagery of the physicalobject. In some embodiments, compositing uses alpha channels associatedwith the imagery of the virtual environment and the imagery of thephysical object. In some embodiments, presenting the virtual environmentwith the isolated imagery of the physical object includes aligning auser perspective of the physical object with a user perspective of thevirtual environment.

In some embodiments, the method further includes: providing, using oneor more speakers, virtual environment audio associated with the virtualenvironment; and in response to detecting that the proximity of theelectronic device to the physical object is less than the firstthreshold distance: providing a combined mix of the virtual environmentaudio with real environment audio, wherein an amount of virtualenvironment audio in the combined mix is associated with the proximityof the electronic device to the physical object. In some embodiments,the virtual environment audio comprises a plurality of audio objects,and wherein providing the combined mix comprises cross-fading theplurality of audio objects with the real environment audio. In someembodiments, an amount of cross-fade applied to one or more first audioobjects of the plurality of audio objects is associated with aprominence of the one or more first audio objects in the virtualenvironment.

In some embodiments, presenting the virtual environment with theisolated imagery of the physical object includes compositing imagery ofthe virtual environment with the isolated imagery of the physical objectusing alpha channels associated with the imagery of the virtualenvironment and the isolated imagery of the physical object, and theamount of cross-fade applied to one or more second audio objects of theplurality of audio objects is associated with values of respective alphachannels.

In accordance with some embodiments, a non-transitory computer-readablestorage medium storing one or more programs configured to be executed byone or more processors of an electronic device is described. The one ormore programs include instructions for: presenting, using the one ormore displays, a virtual environment; detecting a proximity of theelectronic device to a physical object located in a real environment; inresponse to detecting that the proximity of the electronic device to thephysical object is less than a first threshold distance: isolatingimagery of the physical object from other imagery of the realenvironment; and presenting the virtual environment with the isolatedimagery of the physical object inserted into the virtual environment ata location corresponding to the location of the physical object in thereal environment, the imagery of the physical object having a firstvisibility value associated with the proximity of the electronic deviceto the physical object; and in response to detecting that the proximityof the electronic device to the physical object is greater than thefirst threshold distance: presenting the virtual environment without theisolated imagery of the physical object.

In accordance with some embodiments, a transitory computer-readablestorage medium storing one or more programs configured to be executed byone or more processors of an electronic device is described. The one ormore programs include instructions for: presenting, using the one ormore displays, a virtual environment; detecting a proximity of theelectronic device to a physical object located in a real environment; inresponse to detecting that the proximity of the electronic device to thephysical object is less than a first threshold distance: isolatingimagery of the physical object from other imagery of the realenvironment; presenting the virtual environment with the isolatedimagery of the physical object inserted into the virtual environment ata location corresponding to the location of the physical object in thereal environment, the imagery of the physical object having a firstvisibility value associated with the proximity of the electronic deviceto the physical object; and in response to detecting that the proximityof the electronic device to the physical object is greater than thefirst threshold distance: presenting the virtual environment without theisolated imagery of the physical object.

In accordance with some embodiments, an electronic device comprising oneor more processors and memory storing one or more programs configured tobe executed by the one or more processors is described. The one or moreprograms include instructions for: presenting, using the one or moredisplays, a virtual environment; detecting a proximity of the electronicdevice to a physical object located in a real environment; in responseto detecting that the proximity of the electronic device to the physicalobject is less than a first threshold distance: isolating imagery of thephysical object from other imagery of the real environment; presentingthe virtual environment with the isolated imagery of the physical objectinserted into the virtual environment at a location corresponding to thelocation of the physical object in the real environment, the imagery ofthe physical object having a first visibility value associated with theproximity of the electronic device to the physical object; and inresponse to detecting that the proximity of the electronic device to thephysical object is greater than the first threshold distance: presentingthe virtual environment without the isolated imagery of the physicalobject.

In accordance with some embodiments, an electronic device is described.The electronic device includes means for presenting a virtualenvironment; means for detecting a proximity of the electronic device toa physical object located in a real environment; means for, in responseto detecting that the proximity of the electronic device to the physicalobject is less than a first threshold distance: isolating imagery of thephysical object from other imagery of the real environment; presentingthe virtual environment with the isolated imagery of the physical objectinserted into the virtual environment at a location corresponding to thelocation of the physical object in the real environment, the imagery ofthe physical object having a first visibility value associated with theproximity of the electronic device to the physical object; and meansfor, in response to detecting that the proximity of the electronicdevice to the physical object is greater than the first thresholddistance: presenting the virtual environment without the isolatedimagery of the physical object.

DESCRIPTION OF THE FIGURES

FIGS. 1A-1B depict exemplary systems for use in various enhanced realitytechnologies.

FIG. 2 illustrates an example of a real environment where a virtualenvironment is to be viewed, in accordance with some embodiments.

FIG. 3 illustrates an example of a virtual environment, in accordancewith some embodiments.

FIGS. 4A-4E illustrate an exemplary technique for transitioning betweena virtual environment and a real environment, in accordance with someembodiments.

FIGS. 5A-5C illustrate an exemplary technique for inserting imagery froma real environment into a virtual environment, in accordance with someembodiments.

FIG. 6 is a flow diagram illustrating an exemplary process performed byan electronic device, in accordance with some embodiments.

FIG. 7 is a flow diagram illustrating an exemplary process performed byan electronic device, in accordance with some embodiments.

FIG. 8 is a flow diagram illustrating an exemplary process performed byan electronic device, in accordance with some embodiments.

The embodiments depicted in the figures are only exemplary. One skilledin the art will readily recognize from the following discussion thatalternative embodiments of the structures and methods illustrated hereincan be employed without departing from the principles described herein.

DETAILED DESCRIPTION

Various examples of electronic systems and techniques for using suchsystems in relation to various enhanced reality technologies aredescribed.

A physical setting (also referred to as a real environment) refers to aworld with which various persons can sense and/or interact without useof electronic systems. Physical settings, such as a physical park,include physical elements (also referred to as physical objects), suchas, for example, physical wildlife, physical trees, and physical plants.Persons can directly sense and/or otherwise interact with the physicalsetting, for example, using one or more senses including sight, smell,touch, taste, and hearing.

An enhanced reality (ER) setting (also referred to as a virtualenvironment), in contrast to a physical setting, refers to an entirely(or partly) computer-produced setting that various persons, using anelectronic system, can sense and/or otherwise interact with. In ER, aperson's movements are in part monitored, and, responsive thereto, atleast one attribute corresponding to at least one virtual object in theER setting is changed in a manner that is consistent with one or morephysical laws. For example, in response to an ER system detecting aperson looking upward, the ER system may adjust various audio andgraphics presented to the person in a manner consistent with how suchsounds and appearances would change in a physical setting. Adjustmentsto attribute(s) of virtual object(s) in an ER setting also may be made,for example, in response to representations of movement (e.g., voicecommands).

A person may sense and/or interact with an ER object using one or moresenses, such as sight, smell, taste, touch, and sound. For example, aperson may sense and/or interact with objects that create amulti-dimensional or spatial acoustic setting. Multi-dimensional orspatial acoustic settings provide a person with a perception of discreteacoustic sources in multi-dimensional space. Such objects may alsoenable acoustic transparency, which may selectively incorporate audiofrom a physical setting, either with or without computer-produced audio.In some ER settings, a person may sense and/or interact with onlyacoustic objects.

Virtual reality (VR) is one example of ER. A VR setting refers to anenhanced setting that is configured to only include computer-producedsensory inputs for one or more senses. A VR setting includes a pluralityof virtual objects that a person may sense and/or interact with. Aperson may sense and/or interact with virtual objects in the VR settingthrough a simulation of at least some of the person's actions within thecomputer-produced setting, and/or through a simulation of the person orher presence within the computer-produced setting.

Mixed reality (MR) is another example of ER. An MR setting refers to anenhanced setting that is configured to integrate computer-producedsensory inputs (e.g., virtual objects) with sensory inputs from thephysical setting, or a representation of sensory inputs from thephysical setting. On a reality spectrum, an MR setting is between, butdoes not include, a completely physical setting at one end and a VRsetting at the other end.

In some MR settings, computer-produced sensory inputs may be adjustedbased on changes to sensory inputs from the physical setting. Moreover,some electronic systems for presenting MR settings may detect locationand/or orientation with respect to the physical setting to enableinteraction between real objects (i.e., physical elements from thephysical setting or representations thereof) and virtual objects. Forexample, a system may detect movements and adjust computer-producedsensory inputs accordingly, so that, for example, a virtual tree appearsfixed with respect to a physical structure.

Augmented reality (AR) is an example of MR. An AR setting refers to anenhanced setting where one or more virtual objects are superimposed overa physical setting (or representation thereof). As an example, anelectronic system may include an opaque display and one or more imagingsensors for capturing video and/or images of a physical setting. Suchvideo and/or images may be representations of the physical setting, forexample. The video and/or images are combined with virtual objects,wherein the combination is then displayed on the opaque display. Thephysical setting may be viewed by a person, indirectly, via the imagesand/or video of the physical setting. The person may thus observe thevirtual objects superimposed over the physical setting. When a systemcaptures images of a physical setting, and displays an AR setting on anopaque display using the captured images, the displayed images arecalled a video pass-through. Alternatively, a transparent orsemi-transparent display may be included in an electronic system fordisplaying an AR setting, such that an individual may view the physicalsetting directly through the transparent or semi-transparent displays.Virtual objects may be displayed on the semi-transparent or transparentdisplay, such that an individual observes virtual objects superimposedover a physical setting. In yet another example, a projection system maybe utilized in order to project virtual objects onto a physical setting.For example, virtual objects may be projected on a physical surface, oras a holograph, such that an individual observes the virtual objectssuperimposed over the physical setting.

An AR setting also may refer to an enhanced setting in which arepresentation of a physical setting is modified by computer-producedsensory data. As an example, at least a portion of a representation of aphysical setting may be graphically modified (e.g., enlarged), so thatthe modified portion is still representative of (although not afully-reproduced version of) the originally captured image(s).Alternatively, in providing video pass-through, one or more sensorimages may be modified in order to impose a specific viewpoint differentthan a viewpoint captured by the image sensor(s). As another example,portions of a representation of a physical setting may be altered bygraphically obscuring or excluding the portions.

Augmented virtuality (AV) is another example of MR. An AV setting refersto an enhanced setting in which a virtual or computer-produced settingintegrates one or more sensory inputs from a physical setting. Suchsensory input(s) may include representations of one or morecharacteristics of a physical setting. A virtual object may, forexample, incorporate a color associated with a physical element capturedby imaging sensor(s). Alternatively, a virtual object may adoptcharacteristics consistent with, for example, current weather conditionscorresponding to a physical setting, such as weather conditionsidentified via imaging, online weather information, and/orweather-related sensors. As another example, an AR park may includevirtual structures, plants, and trees, although animals within the ARpark setting may include features accurately reproduced from images ofphysical animals.

Various systems allow persons to sense and/or interact with ER settings.For example, a head mounted system may include one or more speakers andan opaque display. As another example, an external display (e.g., asmartphone) may be incorporated within a head mounted system. The headmounted system may include microphones for capturing audio of a physicalsetting, and/or image sensors for capturing images/video of the physicalsetting. A transparent or semi-transparent display may also be includedin the head mounted system. The semi-transparent or transparent displaymay, for example, include a substrate through which light(representative of images) is directed to a person's eyes. The displaymay also incorporate LEDs, OLEDs, liquid crystal on silicon, a laserscanning light source, a digital light projector, or any combinationthereof. The substrate through which light is transmitted may be anoptical reflector, holographic substrate, light waveguide, opticalcombiner, or any combination thereof. The transparent orsemi-transparent display may, for example, transition selectivelybetween a transparent/semi-transparent state and an opaque state. Asanother example, the electronic system may be a projection-based system.In a projection-based system, retinal projection may be used to projectimages onto a person's retina. Alternatively, a projection-based systemalso may project virtual objects into a physical setting, for example,such as projecting virtual objects as a holograph or onto a physicalsurface. Other examples of ER systems include windows configured todisplay graphics, headphones, earphones, speaker arrangements, lensesconfigured to display graphics, heads up displays, automotivewindshields configured to display graphics, input mechanisms (e.g.,controllers with or without haptic functionality), desktop or laptopcomputers, tablets, or smartphones.

FIG. 1A and FIG. 1B depict exemplary system 100 for use in variousenhanced reality technologies.

In some examples, as illustrated in FIG. 1A, system 100 includes device100 a. Device 100 a includes various components, such as processor(s)102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108,orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116,speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122.These components optionally communicate over communication bus(es) 150of device 100 a.

In some examples, elements of system 100 are implemented in a basestation device (e.g., a computing device, such as a remote server,mobile device, or laptop) and other elements of system 100 areimplemented in a second device (e.g., a head-mounted device). In someexamples, device 100 a is implemented in a base station device or asecond device.

As illustrated in FIG. 1B, in some examples, system 100 includes two (ormore) devices in communication, such as through a wired connection or awireless connection. First device 100 b (e.g., a base station device)includes processor(s) 102, RF circuitry(ies) 104, and memory(ies) 106.These components optionally communicate over communication bus(es) 150of device 100 b. Second device 100 c (e.g., a head-mounted device)includes various components, such as processor(s) 102, RF circuitry(ies)104, memory(ies) 106, image sensor(s) 108, orientation sensor(s) 110,microphone(s) 112, location sensor(s) 116, speaker(s) 118, display(s)120, and touch-sensitive surface(s) 122. These components optionallycommunicate over communication bus(es) 150 of device 100 c.

System 100 includes processor(s) 102 and memory(ies) 106. Processor(s)102 include one or more general processors, one or more graphicsprocessors, and/or one or more digital signal processors. In someexamples, memory(ies) 106 are one or more non-transitorycomputer-readable storage mediums (e.g., flash memory, random accessmemory) that store computer-readable instructions configured to beexecuted by processor(s) 102 to perform the techniques described below.

System 100 includes RF circuitry(ies) 104. RF circuitry(ies) 104optionally include circuitry for communicating with electronic devices,networks, such as the Internet, intranets, and/or a wireless network,such as cellular networks and wireless local area networks (LANs). RFcircuitry(ies) 104 optionally includes circuitry for communicating usingnear-field communication and/or short-range communication, such asBluetooth®.

System 100 includes display(s) 120. Display(s) 120 may have an opaquedisplay. Display(s) 120 may have a transparent or semi-transparentdisplay that may incorporate a substrate through which lightrepresentative of images is directed to an individual's eyes. Display(s)120 may incorporate LEDs, OLEDs, a digital light projector, a laserscanning light source, liquid crystal on silicon, or any combination ofthese technologies. The substrate through which the light is transmittedmay be a light waveguide, optical combiner, optical reflector,holographic substrate, or any combination of these substrates. In oneexample, the transparent or semi-transparent display may transitionselectively between an opaque state and a transparent orsemi-transparent state. Other examples of display(s) 120 include headsup displays, automotive windshields with the ability to displaygraphics, windows with the ability to display graphics, lenses with theability to display graphics, tablets, smartphones, and desktop or laptopcomputers. Alternatively, system 100 may be designed to receive anexternal display (e.g., a smartphone). In some examples, system 100 is aprojection-based system that uses retinal projection to project imagesonto an individual's retina or projects virtual objects into a physicalsetting (e.g., onto a physical surface or as a holograph).

In some examples, system 100 includes touch-sensitive surface(s) 122 forreceiving user inputs, such as tap inputs and swipe inputs. In someexamples, display(s) 120 and touch-sensitive surface(s) 122 formtouch-sensitive display(s).

System 100 includes image sensor(s) 108. Image sensors(s) 108 optionallyinclude one or more visible light image sensor, such as charged coupleddevice (CCD) sensors, and/or complementary metal-oxide-semiconductor(CMOS) sensors operable to obtain images of physical elements from thephysical setting. Image sensor(s) also optionally include one or moreinfrared (IR) sensor(s), such as a passive IR sensor or an active IRsensor, for detecting infrared light from the physical setting. Forexample, an active IR sensor includes an IR emitter, such as an IR dotemitter, for emitting infrared light into the physical setting. Imagesensor(s) 108 also optionally include one or more event camera(s)configured to capture movement of physical elements in the physicalsetting. Image sensor(s) 108 also optionally include one or more depthsensor(s) configured to detect the distance of physical elements fromsystem 100. In some examples, system 100 uses CCD sensors, eventcameras, and depth sensors in combination to detect the physical settingaround system 100. In some examples, image sensor(s) 108 include a firstimage sensor and a second image sensor. The first image sensor and thesecond image sensor are optionally configured to capture images ofphysical elements in the physical setting from two distinctperspectives. In some examples, system 100 uses image sensor(s) 108 toreceive user inputs, such as hand gestures. In some examples, system 100uses image sensor(s) 108 to detect the position and orientation ofsystem 100 and/or display(s) 120 in the physical setting. For example,system 100 uses image sensor(s) 108 to track the position andorientation of display(s) 120 relative to one or more fixed elements inthe physical setting.

In some examples, system 100 includes microphones(s) 112. System 100uses microphone(s) 112 to detect sound from the user and/or the physicalsetting of the user. In some examples, microphone(s) 112 includes anarray of microphones (including a plurality of microphones) thatoptionally operate in tandem, such as to identify ambient noise or tolocate the source of sound in space of the physical setting.

System 100 includes orientation sensor(s) 110 for detecting orientationand/or movement of system 100 and/or display(s) 120. For example, system100 uses orientation sensor(s) 110 to track changes in the positionand/or orientation of system 100 and/or display(s) 120, such as withrespect to physical elements in the physical setting. Orientationsensor(s) 110 optionally include one or more gyroscopes and/or one ormore accelerometers.

With reference now to FIGS. 2, 3, 4A-4E, 5A-5C, 6, 7, and 8 , exemplarytechniques for transitioning between imagery and sounds of a virtualenvironment and imagery and sounds of a real environment are described,as well as techniques for inserting imagery from a real environment intoa virtual environment. The transition and/or insertion occurs, in someexamples, in response to an electronic device detecting an event, suchas a signal from an input device, proximity to a physical object (alsoreferred to as a physical element), and/or a triggering sound. Thetechniques enhance user convenience and provide the user with anenhanced degree of awareness by, in some exemplary embodiments,providing imagery and sounds of the real environment at a user's requestand/or in response to an obstacle or alert in the real environment.

FIG. 2 illustrates an example of a real environment 200 (also referredto as a physical setting) where a virtual environment (also referred toas an ER setting) is to be viewed, in accordance with some embodiments.Real environment 200 includes physical objects 202 (also referred to asphysical elements), such as table 202-a, computer 202-b, and walls202-c. While real environment 200 is shown as a room having physicalobjects 202 in FIG. 2 , it should be understood that real environment200 can be any real-world location where a virtual environment is to beviewed.

Real environment 200 is visible to a user of device 100 a, as describedin reference to FIGS. 1A-1B. In some embodiments, real environment 200is displayed to the user by way of a video pass-through mode of device100 a. In other embodiments, the user is provided a substantially directview of real environment 200, such as with a heads-up display.

FIG. 3 illustrates an example of a virtual environment 300 (alsoreferred to as an ER setting), in accordance with some embodiments.Virtual environment 300 includes virtual objects 302, such as virtualrobot 302-a, virtual planet 302-b, and virtual spaceship 302-c. Whilevirtual environment 300 is shown as a deck of a spaceship in FIG. 3 , itshould be understood that virtual environment 300 can be any environmentwhere one or more virtual objects are displayed.

In some examples, during operation, device 100 a (as described inreference to FIGS. 1A-1B) displays virtual environment 300 to a user ofdevice 100 a using display(s) 120. Device 100 a also provides audioassociated with virtual environment 300 to the user using speaker(s)118. The virtual environment audio includes one or more audio objectsassociated with individual components of virtual environment 300. Forexample, first audio object(s) may be associated with virtual robot302-a (e.g., vocalizations, beeping, etc.), second audio object(s) maybe associated with virtual spaceship 302-c (e.g., engine sounds, alarms,etc.), and third audio object(s) may be associated with the ambientnoise of virtual environment 300 (e.g., engine sounds, beeping, etc.).Each of the audio objects are mixed together to form the virtualenvironment audio. In some embodiments, mixing the audio objectsincludes adjusting the volume level, spatial placement, and/or frequencyspectrum of each audio object such that each audio object is blendedinto the virtual environment audio. In some embodiments, the volumelevel, spatial placement, and/or frequency spectrum of each audio objectis adjusted based on the location of an associated virtual object 302 inthe virtual environment 300 and the location and orientation of theuser's head relative to the virtual object 302. In this way, when theuser hears the virtual environment audio, the sounds corresponding toeach audio object appear to be emitting from the virtual location of theassociated virtual object 302 in the virtual environment.

While virtual environment 300 is displayed and virtual environment audiois provided, device 100 a detects movement of the user, includingdetecting a location of the user within real environment 200 and anorientation of the user's head (e.g., where the user is looking). As theuser moves, the view of virtual environment 300 changes to correspond tothe current location and direction of sight of the user. In addition,the mix of audio objects being provided changes to correspond to thecurrent location and position of the user's ears. For example, thevolume level, spatial placement, and/or frequency spectrum or each audioobject changes such that the sounds appear to be emitting from aconsistent virtual location as the user's location and/or positionchanges.

FIGS. 4A-4E illustrate an exemplary technique for transitioning betweenvirtual environment 300 (as described in reference to FIG. 3 ) and realenvironment 200 (as described in reference to FIG. 2 ), in accordancewith some embodiments. As shown in FIG. 4A, a view of virtualenvironment 300 begins to transition to a view of real environment 200.When the transition begins, the view of real environment 200 is at leastpartially visible at the same time that a view of virtual environment300 is provided (e.g., real environment 200 and virtual environment 300are overlaid on one another). In addition, in some embodiments, audiofrom the real environment is at least partially audible when the view ofreal environment 200 becomes at least partially visible.

When transitioning from the view of virtual environment 300 to the viewof real environment 200, imagery of virtual environment 300 iscomposited (e.g., blended) with imagery of real environment 200. Thecompositing is performed by using visibility values associated with theimagery of each environment to combine the environments with each other.In some embodiments, the visibility values correspond to alpha channelinformation in the imagery of each environment. The visibility valuesare used to adjust the transparency of each environment. Before thetransition, the virtual environment 300 has no transparency (e.g.,alpha=1.0). As the transition begins (such as shown in FIG. 4A), thetransparency of the virtual environment 300 is increased (e.g.,alpha=0.9), and imagery of the real environment 200 is added topartially transparent imagery of the virtual environment 300, where thereal environment 200 has a complementary visibility value (e.g.,alpha=0.1).

In some embodiments, the compositing is performed using differentvisibility values for different virtual objects in the virtualenvironment 300. For example, as a transition begins, a first virtualobject (e.g., virtual robot 302-a) has a first visibility value (e.g.,alpha=0.8) while a second virtual object (e.g., virtual spaceship 302-c)has a second, different visibility value (e.g., alpha=0.9). Thesevirtual objects are added to imagery of real environment 200, whereoverlapping portions of the imagery of real environment 200 havecomplementary visibility values (e.g., alpha=0.2 and alpha=0.1,respectively). This allows different portions of the virtual environment300 to fade in or out faster or slower than other portions of thevirtual environment 300. In some embodiments, the visibility valuescorrespond to alpha channel information of individual pixels of thevirtual environment 300 and the real environment 200.

In some embodiment, the compositing is performed using differentvisibility values for different objects in the real environment 200. Forexample, the objects defining the outlines of the real environment 200(e.g., floor, walls, ceiling, windows, desks) may fade in faster thanother objects in the real environment 200 (e.g., objects on the desk,artwork on the walls).

In some embodiments, when the views of virtual environment 300 and realenvironment 200 are combined, the perspective from which the user seesone or both of the environments is shifted. For example, when the viewof real environment 200 is provided using camera(s) that pass-throughthe imagery of real environment, the perspective provided by thecamera(s) may be different than the perspective from which the user seesvirtual environment 300 (e.g., the camera(s)'s perspective is two inchesin front of the user's eyes). Thus, device 100 a shifts the perspectiveof either the real environment 200 or the virtual environment 300 sothat the two environments align. For example, device 100 a modifies theimagery of real environment 200 such that the apparent perspective fromwhich the user views the real environment 200 aligns with theperspective of the virtual environment 300 (e.g., the perspective of thereal environment 200 is shifted back two inches). Alternatively, in someexamples, device 100 a shifts the perspective of the virtual environment300 such that the perspective of the virtual environment 300 aligns withthe perspective of the camera(s) capturing the real environment 200(e.g., the perspective of the virtual environment 300 is shifted forwardtwo inches).

In some embodiments, volume levels of the audio objects associated withvirtual objects 302 in virtual environment 300 change as the view ofvirtual environment 300 transitions to the view of real environment 200.In some embodiments, the volume level of certain audio objects change ata different rate than other audio objects. For example, the volume levelof prominent audio object(s) (e.g., audio associated with a maincharacter, such as virtual robot 302-a) may be reduced more quickly thanthe volume level of audio object(s) used for ambient background noise,or vice versa.

In some embodiments, the volume levels of the audio objects areassociated with visibility values of the virtual and real environments.Thus, when virtual environment 300 becomes less visible, the volumelevel of audio objects associated with virtual environment 300 decreasea corresponding amount. For example, as the visibility value of virtualenvironment 300 is decreased (e.g., alpha changes from 1.0 to 0.9), thevolume levels of the audio objects associated with virtual environment300 also decrease (e.g., −6 dB). The association between the volumelevels of the audio objects and the visibility values of the virtual andreal environments can use various algorithms. In the example above, a10% change in visibility value corresponds to a 6 dB decrease in volume.However, other ratios, functions, or algorithms can be used with thevisibility values to adjust the volume levels of the audio objects. Inaddition, as described above, in some embodiments, the volume level ofcertain audio objects change at a different rate than other audioobjects. In these embodiments, different ratios, functions, oralgorithms are used to adjust the volume level of each audio objectbased on the corresponding visibility values. For example, when thevisibility value of the virtual environment decreases 10% (e.g., alphachanges from 1.0 to 0.9), the volume level of prominent audio object(s)(e.g., audio associated with a main character, such as virtual robot302-a) is reduced by a first amount (e.g., −9 dB), while the volumelevel of less prominent audio object(s) (e.g., ambient background noise)is reduced by a second, different amount (e.g., −6 dB).

In some embodiments, audio objects associated with virtual environment300 are mixed with audio from real environment 200. In some embodiments,the mixing is performed using a loudness transfer function (e.g.,cross-fade) applied to the audio objects associated with virtualenvironment 300 and the audio from real environment 200. The loudnesstransfer function can use various techniques for mixing the audio, suchas a “sin-cos law” in which the crossfade center point is −3 dB (i.e.,0.707 linear). In some embodiments, the audio from real environment 200is detected using microphone(s) 112 of device 100 a, and then mixed withthe audio objects associated with virtual environment based on thevolume level of each respective audio object. In some embodiments, thevolume levels of the audio objects and the audio from real environment200 are associated with the visibility values of the virtual and realenvironments. Thus, in some embodiments, the values for the loudnesstransfer function are associated with the visibility values. Forexample, as the visibility of virtual environment 300 decreases (e.g.,alpha changes from 1.0 to 0.9) and visibility of real environment 200increases (e.g., alpha changes from 0.0 to 0.1), the audio objectsassociated with virtual environment 300 are cross-faded with the audiofrom real environment 200 using loudness transfer function valuescorresponding to the changes in visibility (e.g., volume levels of theaudio objects are decreased and mixed with audio from real environment200 at less than its full volume level). As described above, in someembodiments, the volume levels of certain audio objects (e.g., audioobjects associated with prominent characters in the virtual environment300) change at a different rate than other audio objects (e.g., audioobjects associated with ambient background noise of the virtualenvironment 300). In some embodiments, these different rates of changeto the volume levels are applied when mixing the audio objects with theaudio from real environment 200.

In some embodiments, other forms of audio transitions are used insteadof a cross-fade at the time of the imagery transition. These other audiotransitions include a straight cut (e.g., the audio immediately changeswithout a gradual cross-fade), a pre-lap sound edit (e.g., the audiochanges before the imagery of the environment changes), a post-lap soundedit (e.g., the current audio continues for some time after the imageryof the environment changes), or using the same audio across thetransition (e.g., music or ambient noise from the virtual environment300 continues after the imagery changes to the real environment 200).

In some embodiments, the audio transition includes a transition betweenacoustic properties (e.g., reverberation, decay time (T60), frequencyresponse, time-energy-frequency (TEF), energy time curve (ETC), etc.) ofthe real environment 200 and virtual environment 300. In someembodiments, acoustic properties of audio objects associated withvirtual environment 300 are modified to correspond to the acousticproperties of real environment 200. For example, while in the virtualenvironment 300, audio objects associated with a virtual avatar (e.g.,virtual robot 302-a) include acoustic properties corresponding to thevirtual environment 300 (e.g., sounds emitted by virtual robot 302-a areprocessed to have acoustic properties corresponding to the inside of aspaceship). When the audio transitions to the real environment 200, theacoustic properties of audio objects associated with the virtual avatarare modified to correspond to the acoustic properties of the realenvironment 200 (e.g., the sounds emitted by virtual robot 302-a areprocessed to have acoustic properties corresponding to the room of theuser). In this way, the audio objects associated with virtual avatarblend more realistically with the audio in the real environment 200 whentransitioning from the virtual environment 300 to the real environment200.

In some embodiments, acoustic properties of audio in real environment200 are modified to correspond to the acoustic properties of virtualenvironment 300. For example, while in the real environment 200, thesound of a person speaking in the real environment 200 (e.g., the user'sown voice) has acoustic properties corresponding to the real environment(e.g., the sound of the person speaking is not processed and hasacoustic properties corresponding to the natural acoustics of the roomwhere the person is speaking). When the audio transitions to the virtualenvironment 300, the acoustic properties of audio in the realenvironment (e.g., the sound of the person's voice) are modified tocorrespond to the acoustic properties of the virtual environment 300(e.g., the person's voice is processed to have acoustic propertiescorresponding to the interior of a spaceship). In this way, the audio inthe real environment 200 blends more realistically with the audio in thevirtual environment 300 when transitioning from the real environment 200to the virtual environment 300.

In some embodiments, the acoustic properties of the real environment 200are determined by device 100 a. In some embodiments, the acousticproperties of the real environment 200 are predetermined acousticproperties assigned to a type of environment (e.g., large room, smallroom, concert hall, cathedral). In some embodiments, the acousticproperties of the real environment 200 are determined by analyzingacoustic and/or physical features of the real environment 200. Forexample, the acoustic properties of the real environment 200 can bedetermined by emitting a predetermined test sound (e.g., a chirp orsweep) and analyzing the changes to the test sound caused by the realenvironment 200 (e.g., the test sound reflects off the physical featuresof the room, and the reflections are analyzed). As another example, thephysical features of the room can be detected (e.g., the layout andmaterial composition of the walls, floor, ceiling, and objects in theroom). An acoustic model of the real environment 200 can then beestimated based on the physical features of the real environment 200.

In some embodiments, the audio in the real environment 200 is filteredto remove the acoustic properties of the real environment 200 prior toapplying the acoustic properties of the virtual environment 300. In someembodiments, the acoustic properties of the real environment 200 areprogressively filtered out (e.g., removed gradually) as the acousticproperties of the virtual environment 300 are applied to the audio inthe real environment 200. In this way, the audio in the real environment200 is gradually modified to correspond to the acoustic properties ofthe audio objects of the virtual environment 300 as the view transitionsto the virtual environment 300.

In some embodiments, the acoustic properties applied the audio in thereal environment 200 are based in part on the audio output device (e.g.,closed-back headphones, open-back headphones, in-ear headphones,extra-aural headphones, or loudspeakers). When using an audio outputdevice that allows audio in the real environment 200 to be heard withoutsubstantial distortion or reduction in loudness (e.g., open-backheadphones, extra-aural headphones, loudspeakers), the audio in the realenvironment 200 is not processed (e.g., amplified or filtered) when aview of the real environment 200 is provided. When using an audio outputdevice that causes audio in the real environment 200 to be distorted orreduced in loudness (e.g., closed-back headphones, in-ear headphones),the audio in the real environment 200 is processed (e.g., amplified orfiltered) to apply the acoustic properties of the real environment 200when a view of the real environment 200 is provided. In this way, thedistortion and/or reduction in loudness is mitigated (e.g., a user hearsthe audio in the real environment 200 in a similar way as a user who isnot using closed-back or in-ear headphones).

In some examples, the transition to the view of real environment 200 isin response to electronic device 100 a detecting a transition event. Insome embodiments, detecting the transition event includes receiving asignal from an input device, such as a rotational input mechanism (e.g.,a knob) of the electronic device 100 a. While displaying virtualenvironment 300, if rotation of the rotational input mechanism in afirst direction is detected, then real environment 200 becomes at leastpartially visible and virtual environment 300 becomes less visible(e.g., the environments are blended together as described above). Thevisibility of each environment is associated with an amount of rotation(and/or speed of rotation) of the rotational input mechanism. A largeramount of rotation (and/or faster rotation) results in real environment200 becoming more visible and virtual environment 300 becoming lessvisible than a comparatively small amount of rotation (and/or slowrotation) of the rotational input mechanism. In other words, visibilityvalues used for blending the two environments are associated with theamount of rotation (and/or speed of rotation) of the rotational inputmechanism. This allows a user to control the visibility of eachenvironment (real and virtual). In some embodiments, rotation of therotational input mechanism has a non-linear relationship with thevisibility values. For example, quickly rotating the rotational inputmechanism a half turn results in different visibility values for theenvironments than slowly rotating the rotational input mechanism a halfturn.

In some embodiments, detecting the transition event includes detectingthat a proximity of the electronic device 100 a (and/or the user) to aphysical object (also referred to as a physical element) in realenvironment 200 is less than a threshold distance (e.g., 1 foot). Forexample, while displaying virtual environment 300, if the electronicdevice 100 a (and/or the user) approaches table 202-a of realenvironment 200, real environment 200 becomes at least partially visibleand virtual environment 300 becomes less visible (e.g., the environmentsare blended together as described above). In some examples, visibilityof each environment is associated with the proximity of the device 100 a(and/or the user) to the physical object (e.g., table 202-a). Forinstance, as the device 100 a (and/or the user) moves closer to thephysical object, real environment 200 becomes more visible and virtualenvironment 300 becomes less visible. This provides the user with anindication or warning before they contact the physical object.

In some embodiments, device 100 a detects the proximity of the device100 a and/or the user to a physical object via one or more internaland/or external image sensors. In some embodiments, the thresholddistance is predefined (e.g., by an operating system of device 100 a orset by a user of device 100 a).

In some embodiments, detecting the transition event includes detecting atriggering sound, such as a person speaking or an alert sound (phonecall, message, alarm, etc.). For example, while displaying virtualenvironment 300, if a triggering sound is detected, real environment 200becomes at least partially visible and virtual environment 300 becomesless visible (e.g., the environments are blended together as describedabove). By providing an at least partially visible view of realenvironment 200, the user may identify the source of the triggeringsound. In some embodiments, a representation of the source of thetriggering sound (e.g., the person speaking, a phone, a computer, etc.)is more visible than other elements of real environment 200.

As shown in FIG. 4B, the transition from the view of virtual environment300 to the view of real environment 200 continues by real environment200 becoming more visible. In some embodiments, the transition continuesin response to further input on the input device, such as the rotationalinput mechanism. In some embodiments, the transition continues inresponse to detecting that the proximity of device 100 a to the physicalobject in real environment 200 has decreased. Thus, as the userapproaches the physical object, the physical object (and realenvironment 200) becomes more visible. In addition, in some embodiments,the volume levels of audio objects associated with virtual environment300 continue to be reduced, while audio associated with real environment200 is provided. In some embodiments, the volume level of certain audioobjects (e.g., audio objects associated with prominent virtual objects,such as virtual robot 302-a) is reduced more quickly than other audioobjects (e.g., audio objects associated with ambient background noise ofvirtual environment 300), or vice versa.

As shown in FIG. 4C, the view of virtual environment 300 hastransitioned approximately half way to the view of real environment,such that both environments are approximately equally visible. Inaddition, in some embodiments, audio objects associated with virtualenvironment 300 are provided at approximately the same volume levels asaudio associated with real environment 200.

As shown in FIG. 4D, the transition from the view of virtual environment300 to the view of real environment 200 continues by virtual environment300 becoming less visible than real environment 200.

As shown in FIG. 4E, the transition from the view of virtual environment300 to the view of real environment 200 continues by virtual environment300 continuing to become less visible than real environment 200.

When the transition is complete, device 100 a provides a view of realenvironment 200 as shown in FIG. 2 , without providing a view of virtualenvironment 300.

While providing a view of real environment 200, a substantially oppositeprocess can be performed to transition from the view of real environment200 to a view of virtual environment 300. This process is performed inresponse to detecting another transition event. In some embodiments, thetransition event includes receiving a signal from an input device (e.g.,the rotational input mechanism in an opposite direction) or detecting aproximity of device 100 a to a physical object is greater than athreshold distance (e.g., 1 foot).

Furthermore, during the transition process, the transition can bereversed so that the view of virtual environment 300 becomes morevisible while the view of real environment 200 becomes less visible (orvice versa). The reversal of the transition process is in response todetecting a second transition event. In some embodiments, detecting thesecond transition event includes receiving a second signal from theinput device (e.g., the rotational input mechanism). For example, if auser begins rotating the rotational input mechanism in a first directionwhich causes a transition from the view of virtual environment 300 tothe view of real environment 200, then a rotation of the rotationalinput mechanism in a second, opposite direction causes the transition toreverse. In some embodiments, the rotation of the rotational inputmechanism in the second, opposite direction has a non-linearrelationship with the visibility values of the environments. Forexample, quickly rotating the rotational input mechanism a half turnresults in different visibility values for the environments than slowlyrotating the rotational input mechanism a half turn.

The reversal of the transition can occur at various times during thetransition process. For example, if the transition is partiallycomplete, as shown in FIG. 4B, and a second transition event isdetected, the transition process reverses and the view of the realenvironment becomes less visible, as shown in FIG. 4A.

In some embodiments, detecting the second transition event includesdetecting that the proximity of device 100 a to the physical object hasincreased. In response to this increase in proximity, the transitionprocess reverses as described above.

This process of transitioning back and forth between views of virtualenvironment 300 and views of real environment 200 provides a user withvisual (and in some embodiments, audible) cues that they approaching anobstacle without fully interrupting the user's experience in the virtualenvironment 300. By gradually fading in the view of real environment200, the user may be less disoriented than if virtual environment 300were suddenly replaced with a view of real environment 200. Then, whenthe user is no longer near the obstacle, the view of real environment200 fades out and the user's experience in the virtual environment 300proceeds more naturally than if than the virtual environment 300suddenly reappeared.

FIGS. 5A-5C illustrate an exemplary technique for inserting imagery fromreal environment 200 (as described in reference to FIG. 2 ) into virtualenvironment 300 (as described in reference to FIG. 3 ), in accordancewith some embodiments. As shown in FIG. 5A, imagery of physical object202-a (also referred to as a physical element) (e.g., a table) isinserted into virtual environment 300. When the imagery of physicalobject 202-a is initially inserted into virtual environment 300, theimagery of physical object 202-a is at least partially visible.

The imagery of physical object 202-a is inserted in response todetecting that a proximity of the electronic device 100 a (and/or theuser) to the physical object 202-a in real environment 200 is less thana threshold distance (e.g., 1 foot). For example, while displayingvirtual environment 300, if the electronic device 100 a approachesphysical object 202-a in real environment 200, imagery of physicalobject 202-a becomes at least partially visible within virtualenvironment 300 at a location corresponding to the real-world locationof the physical object 202-a in real environment 200. In someembodiments, imagery of virtual environment 300 (e.g., at the locationcorresponding to imagery of physical object 202-a) also becomes lessvisible when the imagery of physical object 202-a is inserted.

In some embodiments, device 100 a detects the proximity of the device100 a and/or the user (e.g., arms, legs, or head of the user's body) tothe physical object 202-a via one or more internal and/or external imagesensors. In some embodiments, the threshold distance is predefined(e.g., by an operating system of device 100 a or set by a user of device100 a).

In some embodiments, imagery of physical object 202-a is isolated fromother imagery of real environment 200. For example, when device 100 adetects the proximity to the physical object 202-a is less than thethreshold distance (e.g., 1 foot), imagery of physical object 202-a isisolated from the surrounding imagery of real environment 200. Theisolated imagery of physical object 202-a is then inserted into imageryof virtual environment 300. In some embodiments, the imagery of physicalobject 202-a is detected via one or more sensors (e.g., image sensors)of device 100 a, for instance, when the physical object 202-a is in thefield of vision of the one or more sensors of device 100 a. In someembodiments, the physical object 202-a is additionally or alternativelydetected via one or more sensors (e.g., image sensors, depth sensorsmotion sensors, accelerometers, gyroscopes) external to device 100 a(e.g., sensors installed in real environment 200).

In some embodiments, the imagery of physical object 202-a is depictedwithin virtual environment 300 using a generic object shape (e.g., ageneric table, a cube, a sphere, etc). In some embodiments, the genericobject shape visually resembles the corresponding physical object 202-ain the real environment 200 (e.g., similar size, similar color, etc).

In some embodiments, the imagery of physical object 202-a is composited(e.g., blended) with imagery of virtual environment 300. The compositingis performed by using visibility values associated with the imagery ofthe physical object 202-a and the imagery of virtual environment 300 tocombine the imageries with each other. In some embodiments, thevisibility values correspond to alpha channel information in theimageries. The visibility values are used to adjust the transparency ofthe imagery of the physical object 202-a and the imagery of virtualenvironment 300. Before inserting, the virtual environment 300 has notransparency (e.g., alpha=1.0). When the imagery of physical object isinitially inserted (such as shown in FIG. 5A), the transparency of thevirtual environment 300 where the physical object 202-a is beinginserted is increased slightly (e.g., alpha=0.9), and imagery ofphysical object 202-a is added to the partially transparent imagery ofthe virtual environment 300, where the imagery of physical object 202-ahas a complementary visibility value (e.g., alpha=0.1). In someembodiments, the visibility values correspond to alpha channelinformation of individual pixels of the virtual environment 300 and theimagery of physical object 202-a. In this manner, one or more portionsof the virtual environment 300 may be displayed at a differenttransparency than one or more other portions of the virtual environment300, for instance, before, during, and/or after a transition betweenviews of virtual environment 300 and views of real environment 200.

The visibility of the imagery of physical object 202-a is associatedwith the proximity of the device 100 a (and/or the user) to the physicalobject 202-a. As the device 100 a moves closer to the physical object202-a in real environment 200, the imagery of physical object 202-ainserted into virtual environment 300 becomes more visible. In someembodiments, imagery of virtual environment 300 also becomes lessvisible. This provides the user with an indication or warning beforethey contact the physical object.

As shown in FIG. 5B, the imagery of physical object 202-a becomes morevisible in virtual environment 300 as the proximity of device 100 a(and/or the user) decreases. Furthermore, in some embodiments, imageryof virtual environment 300 at the location where the imagery of physicalobject 202-a is inserted becomes less visible. In some embodiments,imagery of all of virtual environment 300 becomes less visible as theimagery of physical object 202-a becomes more visible.

As shown in FIG. 5C, the imagery of physical object 202-a is fullyvisible (e.g., alpha=1.0) in virtual environment 300. The imagery ofphysical object 202-a becomes fully visible in response to the proximityof device 100 a (and/or the user) decreasing to less than a secondthreshold distance (e.g., 3 inches). Furthermore, in some embodiments,imagery of virtual environment 300 at the location where the imagery ofphysical object 202-a is inserted is no longer visible (e.g.,alpha=0.0). In some embodiments, imagery of all of virtual environment300 is no longer visible such that only the imagery of physical object202-a is visible when the proximity to the physical object 202-adecreases to less than the second threshold.

After imagery of physical object 202-a is inserted in virtualenvironment 300, a substantially opposite process can be performed toremove the imagery of physical object 202-a from virtual environment300. This reversal process is in response to detecting the proximity ofdevice 100 a (and/or the user) to the physical object 202-a hasincreased to more than the threshold distance. This reversal can occurat various times after the imagery of physical object 202-a becomesvisible in virtual environment 300. For example, if the imagery ofphysical object 202-a is partially visible, as shown in FIG. 5B, and theproximity to the physical object 202-a increases, the imagery ofphysical object 202-a becomes less visible, such as shown in FIG. 5A. Ifthe proximity to the physical object 202-a continues to increase togreater than the threshold distance, then the imagery of physical object202-a is removed from virtual environment 300, and virtual environmentis displayed as shown in FIG. 3 .

This process of inserting imagery of physical object 202-a in virtualenvironment 300 provides a user with visual (and in some embodiments,audible) cues that they approaching an obstacle without fullyinterrupting the user's experience in the virtual environment 300. Byinserting imagery of physical object 202-a in virtual environment 300while keeping other aspects of virtual environment 300 intact, the usermay be less disoriented than if all aspects of virtual environment 300were suddenly replaced. Then, when the user is no longer near theobstacle, the imagery of the physical object 202-a disappears and theuser's experience in the virtual environment 300 proceeds more naturallythan if all aspects of virtual environment 300 suddenly reappeared.

FIG. 6 is a flow diagram illustrating an exemplary process 600 performedby an electronic device (e.g., device 100 a), in accordance with someembodiments. In some embodiments, the electronic device has one or moredisplay(s) (e.g., display(s) 120). In some embodiments, the display is a(at least partially) transparent display. In some embodiments, theelectronic device is connected to and in communication with a displaythat is separate from the device. In some embodiments, the electronicdevice has one or more sensor devices (e.g., image sensor(s) 108,orientation sensor(s) 110, location sensor(s) 116). In some embodiments,the electronic device is connected to and in communication with one ormore sensor devices (e.g., image sensor(s) 108, orientation sensor(s)110, location sensor(s) 116) that are separate from the device. In someembodiments, the device is a head-mounted device. In some embodiments,the electronic device is separate from but is secured on (or configuredto be secured to) a head-mounted device. In some embodiments, theelectronic device includes one or more speakers (e.g., speaker(s) 118)for outputting audio. In some embodiments, the electronic device isconnected (or configured to be connected) to (e.g., via wirelessconnection, via wired connection) and in communication (or configured tobe in communication) with one or more speakers (e.g., speaker(s) 118)for outputting audio.

At block 602, a view of a first environment of a first type is provided.In some embodiments, the first environment of the first type is avirtual environment (e.g., virtual environment 300). In someembodiments, the first environment of the first type is a representationof a real environment (e.g., real environment 200). When the view of thefirst environment is initially provided, the environment is provided ata first visibility value (e.g., alpha=1.0, such as shown in FIG. 3 ).

At block 604, first environment audio associated with the firstenvironment is provided. In some embodiments, the first environmentaudio includes one or more audio objects associated with individualcomponents of a virtual environment (such as virtual environment 300, asshown in FIG. 3 ). Each of the audio objects are mixed together to formthe first environment audio. In some embodiments, mixing the audioobjects includes adjusting the volume level, spatial placement, and/orfrequency spectrum of each audio object such that each audio object isblended into the first environment audio. In some embodiments, thevolume level, spatial placement, and/or frequency spectrum of each audioobject is adjusted based on the location of an associated virtual object(e.g., virtual object 302) in a virtual environment (e.g., virtualenvironment 300) and the location and orientation of the user's headrelative to the virtual object.

At block 606, a transition event is detected. In some embodiments,detecting the transition event includes receiving a signal from an inputdevice. In some embodiments, the input device comprises a rotationalinput mechanism (e.g., a knob). In some embodiments, rotation of therotational input mechanism has a non-linear relationship with thereceived signal (e.g., quickly rotating the knob results in a differentsignal than slowly rotating the knob).

In some embodiments, detecting the transition event includes detectingthat a proximity of a user to a physical object in a real environment(e.g., table 202-a, as shown in FIG. 2 ) is less than a thresholddistance. In some embodiments, detecting the transition event includesdetecting a triggering sound.

At block 608, in response to detecting the transition event, a combinedview of the first environment with a second environment of a second typeis provided (such as shown in FIGS. 4A-4E). The combined view includesimagery of the first environment at a first visibility value and imageryof the second environment at a second visibility value. In someembodiments, the visibility values correspond to alpha channelinformation in the imagery of each environment. The visibility valuesare used to adjust the transparency of each environment. In someembodiments, the first and second visibility values are based on thereceived signal from the input device. In some embodiments, whendetecting the transition event includes detecting that the proximity ofthe user to a physical object in the real environment is less than athreshold distance, the first visibility value and the second visibilityvalue are based on the proximity of the user to the physical object. Insome embodiments, the imagery of the second environment includes arepresentation of the physical object (e.g., imagery of table 202-a). Insome embodiments, when detecting the transition event includes detectinga triggering sound, the imagery of the second environment includes arepresentation of the source of the triggering sound.

In some embodiments, the second environment of the second type is arepresentation of a real environment (e.g., real environment 200). Insome embodiments, the second environment of the second type is a virtualenvironment (e.g., virtual environment 300). In some embodiments,providing the combined view includes compositing the imagery of thefirst environment with the imagery of the second environment. In someembodiments, the compositing uses alpha channels associated with thefirst environment and the second environment.

At block 610, in response to detecting the transition event, a combinedmix of the first environment audio with second environment audioassociated with the second environment is provided. At least one of thefirst environment audio or the second environment audio includes aplurality of audio objects, and volume levels of the plurality of audioobjects are associated with at least one of the first visibility valueor the second visibility value. In some embodiments, the firstenvironment audio includes the plurality of audio objects, and providingthe combined mix includes cross-fading the plurality of audio objectswith the second environment audio in accordance with the respectivevolume levels of the plurality of audio objects. In some embodiments, avolume level of one or more first audio objects of the plurality ofaudio objects is further associated with a prominence of the one or morefirst audio objects in the first environment audio (e.g., audio objectsassociated with a main character of virtual environment 300, such asvirtual robot 302-a). In some embodiments, providing the combined viewincludes aligning a user perspective of the first environment with auser perspective of the second environment.

In some embodiments, after providing the combined view of the firstenvironment with the second environment and the combined mix of thefirst environment audio with the second environment audio, provision ofthe combined view of the first environment with the second environmentand the combined mix of the first environment audio with the secondenvironment audio is ceased. A view of the second environment and secondenvironment audio are then provided (e.g., a view of real environment200 is provided, as shown in FIG. 2 , with audio from the realenvironment 200).

In some embodiments, a second transition event is detected (e.g.,further rotation of the input mechanism, closer proximity to a physicalobject). In response to detecting the second transition event, provisionof the combined view of the first environment with the secondenvironment and the combined mix of the first environment audio with thesecond environment audio is ceased. A view of the second environment andsecond environment audio are then provided (e.g., a view of realenvironment 200 is provided, as shown in FIG. 2 , with audio from thereal environment 200).

In some embodiments, in response to detecting the second transitionevent, the first and second visibility values and the combined mix ofthe first environment audio with the second environment audio aremodified (e.g., the transition from the view of virtual environment 300to the view of real environment 200 continues, as shown in FIGS. 4B-4E).

In some embodiments, in response to detecting the second transitionevent, provision of the combined view of the first environment with thesecond environment and the combined mix of the first environment audiowith the second environment audio is ceased. A view of the firstenvironment and first environment audio are then provided (e.g., theview returns to a view of virtual environment 300, as shown in FIG. 3 ,with audio objects associated with the virtual environment 300).

FIG. 7 is a flow diagram illustrating an exemplary process 700 performedby an electronic device (e.g., device 100 a), in accordance with someembodiments. In some embodiments, the electronic device has one or moredisplay(s) (e.g., display(s) 120). In some embodiments, the display is a(at least partially) transparent display. In some embodiments, theelectronic device is connected to and in communication with a displaythat is separate from the device. In some embodiments, the electronicdevice has one or more sensor devices (e.g., image sensor(s) 108,orientation sensor(s) 110, location sensor(s) 116). In some embodiments,the electronic device is connected to and in communication with one ormore sensor devices (e.g., image sensor(s) 108, orientation sensor(s)110, location sensor(s) 116) that are separate from the device. In someembodiments, the device is a head-mounted device. In some embodiments,the electronic device is separate from but is secured on (or configuredto be secured to) a head-mounted device. In some embodiments, theelectronic device includes one or more speakers (e.g., speaker(s) 118)for outputting audio. In some embodiments, the electronic device isconnected (or configured to be connected) to (e.g., via wirelessconnection, via wired connection) and in communication (or configured tobe in communication) with one or more speakers (e.g., speaker(s) 118)for outputting audio.

At block 702, a virtual environment (e.g., virtual environment 300) ispresented (e.g., displayed) using one or more displays. When virtualenvironment is initially presented, the environment is presented at afirst visibility value (e.g., alpha=1.0, such as shown in FIG. 3 ).

At block 704, proximity of the electronic device to a physical objectlocated in a real environment (e.g., physical object 202-a, as shown inFIG. 2 ) is detected.

At block 706, in response to detecting that the proximity of theelectronic device to the physical object is less than a first thresholddistance, imagery of the physical object is isolated from other imageryof the real environment.

At block 708, the virtual environment is presented with the isolatedimagery of the physical object inserted into the virtual environment ata location corresponding to the location of the physical object in thereal environment (such as shown in FIGS. 5A-5C). The imagery of thephysical object has a first visibility value associated with theproximity of the electronic device to the physical object (e.g., as thedevice moves closer to the physical object 202-a in real environment200, the imagery of physical object 202-a inserted into virtualenvironment 300 becomes more visible). In some embodiments, presentingthe virtual environment with the isolated imagery of the physical objectincludes compositing imagery of the virtual environment with theisolated imagery of the physical object. In some embodiments, thecompositing uses alpha channels associated with the imagery of thevirtual environment and the imagery of the physical object. In someembodiments, presenting the virtual environment with the isolatedimagery of the physical object includes aligning a user perspective ofthe physical object with a user perspective of the virtual environment

In some embodiments, in response to detecting that the proximity of theelectronic device to the physical object is less than the firstthreshold distance, the virtual environment is presented at a secondvisibility value associated with the proximity of the electronic deviceto the physical object (e.g., imagery of virtual environment 300 becomesless visible when the imagery of physical object 202-a is inserted). Insome embodiments, in response to detecting that the proximity of theelectronic device to the physical object is less than a second thresholddistance, the first visibility value of the imagery of the physicalobject is modified (e.g., the imagery of physical object 202-a becomesmore visible as the device moves closer to the object). In someembodiments, in response to detecting that the proximity of theelectronic device to the physical object is less than a third thresholddistance, the presentation of the virtual environment is ceased and aview of the real environment is provided (e.g., a view of realenvironment 200 is provided, as shown in FIG. 2 ).

At block 710, in response to detecting that the proximity of theelectronic device to the physical object is greater than the firstthreshold distance, the virtual environment is presented without theisolated imagery of the physical object (e.g., imagery of physicalobject 202-a is removed and virtual environment 300 is presented, asshown in FIG. 3 ).

In some embodiments, process 700 further includes providing, using oneor more speakers, virtual environment audio associated with the virtualenvironment. In some embodiments, in response to detecting that theproximity of the electronic device to the physical object is less thanthe first threshold distance, a combined mix of the virtual environmentaudio with real environment audio is provided, where an amount ofvirtual environment audio in the combined mix is associated with theproximity of the electronic device to the physical object.

In some embodiments, the virtual environment audio includes a pluralityof audio objects, and providing the combined mix includes cross-fadingthe plurality of audio objects with the real environment audio. In someembodiments, an amount of cross-fade applied to one or more first audioobjects of the plurality of audio objects is associated with aprominence of the one or more first audio objects in the virtualenvironment (e.g., audio objects associated with a main character ofvirtual environment 300, such as virtual robot 302-a). In someembodiments, the amount of cross-fade applied to one or more secondaudio objects of the plurality of audio objects is associated withvalues of alpha channels of the imagery.

FIG. 8 is a flow diagram illustrating an exemplary process 800 performedby an electronic device (e.g., device 100 a), in accordance with someembodiments. In some embodiments, the electronic device has one or moredisplay(s) (e.g., display(s) 120). In some embodiments, the display is a(at least partially) transparent display. In some embodiments, theelectronic device is connected to and in communication with a displaythat is separate from the device. In some embodiments, the electronicdevice has one or more sensor devices (e.g., image sensor(s) 108,orientation sensor(s) 110, location sensor(s) 116). In some embodiments,the electronic device is connected to and in communication with one ormore sensor devices (e.g., image sensor(s) 108, orientation sensor(s)110, location sensor(s) 116) that are separate from the device. In someembodiments, the device is a head-mounted device. In some embodiments,the electronic device is separate from but is secured on (or configuredto be secured to) a head-mounted device. In some embodiments, theelectronic device includes one or more speakers (e.g., speaker(s) 118)for outputting audio. In some embodiments, the electronic device isconnected (or configured to be connected) to (e.g., via wirelessconnection, via wired connection) and in communication (or configured tobe in communication) with one or more speakers (e.g., speaker(s) 118)for outputting audio.

At block 802, a view of a first environment of a first type is provided(e.g., displayed). In some embodiments, the first environment of thefirst type is a real environment (e.g., real environment 200). In someembodiments, the first environment of the first type is a virtualenvironment (e.g., virtual environment 300).

At block 804, first environment audio associated with the firstenvironment is provided, wherein the first environment audio has firstacoustic properties corresponding to the provided view of the firstenvironment. In some embodiments, providing the first environment audioincludes modifying the first environment audio to have the firstacoustic properties corresponding to the provided view of the firstenvironment. In some embodiments, the first environment audio includes auser's voice. In some embodiments, the first environment audio includesan audio object associated with a component of the first environment ofthe first type.

At block 806, a transition event is detected. In some embodiments,detecting the transition event includes receiving a signal from an inputdevice. In some embodiments, the input device includes a rotationalinput mechanism (e.g., a knob). In some embodiments, detecting thetransition event includes detecting that a proximity of a user to aphysical object (e.g., table 202-a) in a real environment is less than athreshold distance. In some embodiments, detecting the transition eventincludes detecting a triggering sound.

At block 808, in response to detecting the transition event, a view of asecond environment of a second type is provided. In some embodiments,the second environment of the second type is a virtual environment(e.g., virtual environment 300). In some embodiments, the secondenvironment of the second type is a real environment (e.g., realenvironment 200).

At block 810, the first environment audio is modified to have secondacoustic properties corresponding to the provided view of the secondenvironment. In some embodiments, the acoustic properties include one ormore of reverberation, decay time, frequency response,time-energy-frequency, and energy time curve.

At block 812, the modified first environment audio is provided. In someembodiments, the modified first environment audio is provided in acombined mix with second environment audio associated with the secondenvironment. In some embodiments, the second environment audio includesaudio modified to have second acoustic properties corresponding to theprovided view of the second environment.

In some embodiments, a second transition event is detected. In responseto detecting the second transition event, the view of the secondenvironment of the second type ceases to be provided, the firstenvironment audio ceases to be modified to have second acousticproperties, a view of the first environment of the first type isprovided, and the first environment audio is provided, wherein the firstenvironment audio has first acoustic properties corresponding to theprovided view of the first environment.

Executable instructions for performing the features of processes 600,700, and 800 described above are, optionally, included in a transitoryor non-transitory computer-readable storage medium (e.g., memory(ies)106) or other computer program product configured for execution by oneor more processors (e.g., processor(s) 102).

Various processes defined herein consider the option of obtaining andutilizing a user's personal information. For example, such personalinformation may be utilized in order to provide imagery and sounds of areal environment in a virtual environment. However, to the extent suchpersonal information is collected, such information should be obtainedwith the user's informed consent. As described herein, the user shouldhave knowledge of and control over the use of their personalinformation.

Personal information will be utilized by appropriate parties only forlegitimate and reasonable purposes. Those parties utilizing suchinformation will adhere to privacy policies and practices that are atleast in accordance with appropriate laws and regulations. In addition,such policies are to be well-established, user-accessible, andrecognized as in compliance with or above governmental/industrystandards. Moreover, these parties will not distribute, sell, orotherwise share such information outside of any reasonable andlegitimate purposes.

Users may, however, limit the degree to which such parties may access orotherwise obtain personal information. For instance, settings or otherpreferences may be adjusted such that users can decide whether theirpersonal information can be accessed by various entities. Furthermore,while some features defined herein are described in the context of usingpersonal information, various aspects of these features can beimplemented without the need to use such information. As an example, ifuser preferences, account names, and/or location history are gathered,this information can be obscured or otherwise generalized such that theinformation does not identify the respective user.

The foregoing descriptions of specific embodiments have been presentedfor purposes of illustration and description. They are not intended tobe exhaustive or to limit the scope of the claims to the precise formsdisclosed, and it should be understood that many modifications andvariations are possible in light of the above teaching.

What is claimed is:
 1. An electronic device, comprising: one or moredisplays; one or more processors; and memory storing one or moreprograms configured to be executed by the one or more processors, theone or more programs including instructions for: presenting, using theone or more displays, a virtual environment; detecting a proximity ofthe electronic device to a physical object located in a realenvironment; in response to detecting that the proximity of theelectronic device to the physical object is less than a first thresholddistance: isolating imagery of the physical object from other imagery ofthe real environment; and presenting the virtual environment with theisolated imagery of the physical object inserted into the virtualenvironment at a location corresponding to the location of the physicalobject in the real environment, the imagery of the physical objecthaving a first visibility value associated with the proximity of theelectronic device to the physical object, wherein presenting the virtualenvironment with the isolated imagery of the physical object includes:presenting a first portion of the virtual environment at a firsttransparency associated with the proximity of the electronic device tothe physical object, wherein presenting the first portion of the virtualenvironment at the first transparency includes combining the firstportion of the virtual environment with first imagery of the realenvironment; and presenting a second portion of the virtual environmentat a second transparency, different from the first transparency,associated with the proximity of the electronic device to the physicalobject, wherein presenting the second portion of the virtual environmentat the second transparency includes combining the second portion of thevirtual environment with second imagery of the real environment; and inresponse to detecting that the proximity of the electronic device to thephysical object is greater than the first threshold distance: presentingthe virtual environment without the isolated imagery of the physicalobject.
 2. The electronic device of claim 1, wherein the one or moreprograms further include instructions for: in response to detecting thatthe proximity of the electronic device to the physical object is lessthan a second threshold distance: modifying the first visibility valueof the imagery of the physical object.
 3. The electronic device of claim1, wherein the one or more programs further include instructions for: inresponse to detecting that the proximity of the electronic device to thephysical object is less than a third threshold distance: ceasing topresent the virtual environment; and providing a view of the realenvironment.
 4. The electronic device of claim 1, wherein presenting thevirtual environment with the isolated imagery of the physical objectcomprises compositing imagery of the virtual environment with theisolated imagery of the physical object, wherein the compositing usesalpha channels associated with the imagery of the virtual environmentand the imagery of the physical object.
 5. The electronic device ofclaim 1, wherein the one or more programs further include instructionsfor: providing, using one or more speakers, virtual environment audioassociated with the virtual environment; and in response to detectingthat the proximity of the electronic device to the physical object isless than the first threshold distance: providing a combined mix of thevirtual environment audio with real environment audio, wherein an amountof virtual environment audio in the combined mix is associated with theproximity of the electronic device to the physical object.
 6. Theelectronic device of claim 5, wherein the virtual environment audiocomprises a plurality of audio objects, and wherein providing thecombined mix comprises cross-fading the plurality of audio objects withthe real environment audio.
 7. The electronic device of claim 6, whereinan amount of cross-fade applied to one or more first audio objects ofthe plurality of audio objects is associated with a prominence of theone or more first audio objects in the virtual environment.
 8. Theelectronic device of claim 6, wherein: presenting the virtualenvironment with the isolated imagery of the physical object comprisescompositing imagery of the virtual environment with the isolated imageryof the physical object using alpha channels associated with the imageryof the virtual environment and the isolated imagery of the physicalobject, and the amount of cross-fade applied to one or more second audioobjects of the plurality of audio objects is associated with values ofrespective alpha channels.
 9. The electronic device of claim 1, whereinpresenting the virtual environment with the isolated imagery of thephysical object includes aligning a user perspective of the physicalobject with a user perspective of the virtual environment.
 10. Theelectronic device of claim 1, wherein the first portion of the virtualenvironment corresponds to the location of the physical object in thereal environment, and wherein the second portion of the virtualenvironment corresponds to a location different from the location of thephysical object in the real environment.
 11. The electronic device ofclaim 1, wherein the first transparency and the second transparencyincrease as the proximity of the electronic device to the physicalobject decreases.
 12. A non-transitory computer-readable storage mediumstoring one or more programs configured to be executed by one or moreprocessors of an electronic device having one or more displays, the oneor more programs including instructions for: presenting, using the oneor more displays, a virtual environment; detecting a proximity of theelectronic device to a physical object located in a real environment; inresponse to detecting that the proximity of the electronic device to thephysical object is less than a first threshold distance: isolatingimagery of the physical object from other imagery of the realenvironment; and presenting the virtual environment with the isolatedimagery of the physical object inserted into the virtual environment ata location corresponding to the location of the physical object in thereal environment, the imagery of the physical object having a firstvisibility value associated with the proximity of the electronic deviceto the physical object, wherein presenting the virtual environment withthe isolated imagery of the physical object includes: presenting a firstportion of the virtual environment at a first transparency associatedwith the proximity of the electronic device to the physical object,wherein presenting the first portion of the virtual environment at thefirst transparency includes combining the first portion of the virtualenvironment with first imagery of the real environment; and presenting asecond portion of the virtual environment at a second transparency,different from the first transparency, associated with the proximity ofthe electronic device to the physical object, wherein presenting thesecond portion of the virtual environment at the second transparencyincludes combining the second portion of the virtual environment withsecond imagery of the real environment; and in response to detectingthat the proximity of the electronic device to the physical object isgreater than the first threshold distance: presenting the virtualenvironment without the isolated imagery of the physical object.
 13. Thenon-transitory computer-readable storage medium of claim 12, wherein theone or more programs further include instructions for: in response todetecting that the proximity of the electronic device to the physicalobject is less than a second threshold distance: modifying the firstvisibility value of the imagery of the physical object.
 14. Thenon-transitory computer-readable storage medium of claim 12, wherein theone or more programs further include instructions for: in response todetecting that the proximity of the electronic device to the physicalobject is less than a third threshold distance: ceasing to present thevirtual environment; and providing a view of the real environment. 15.The non-transitory computer-readable storage medium of claim 12, whereinthe one or more programs further include instructions for: providing,using one or more speakers, virtual environment audio associated withthe virtual environment, wherein the virtual environment audio comprisesa plurality of audio objects; and in response to detecting that theproximity of the electronic device to the physical object is less thanthe first threshold distance: providing a combined mix of the virtualenvironment audio with real environment audio, wherein an amount ofvirtual environment audio in the combined mix is associated with theproximity of the electronic device to the physical object, whereinproviding the combined mix comprises cross-fading the plurality of audioobjects with the real environment audio.
 16. The non-transitorycomputer-readable storage medium of claim 15, wherein: presenting thevirtual environment with the isolated imagery of the physical objectcomprises compositing imagery of the virtual environment with theisolated imagery of the physical object using alpha channels associatedwith the imagery of the virtual environment and the isolated imagery ofthe physical object, and the amount of cross-fade applied to one or moresecond audio objects of the plurality of audio objects is associatedwith values of respective alpha channels.
 17. The non-transitorycomputer-readable storage medium of claim 12, wherein the first portionof the virtual environment corresponds to the location of the physicalobject in the real environment, and wherein the second portion of thevirtual environment corresponds to a location different from thelocation of the physical object in the real environment.
 18. Thenon-transitory computer-readable storage medium of claim 12, wherein thefirst transparency and the second transparency increase as the proximityof the electronic device to the physical object decreases.
 19. A method,comprising: at an electronic device having one or more displays:presenting, using the one or more displays, a virtual environment;detecting a proximity of the electronic device to a physical objectlocated in a real environment; in response to detecting that theproximity of the electronic device to the physical object is less than afirst threshold distance: isolating imagery of the physical object fromother imagery of the real environment; and presenting the virtualenvironment with the isolated imagery of the physical object insertedinto the virtual environment at a location corresponding to the locationof the physical object in the real environment, the imagery of thephysical object having a first visibility value associated with theproximity of the electronic device to the physical object, whereinpresenting the virtual environment with the isolated imagery of thephysical object includes: presenting a first portion of the virtualenvironment at a first transparency associated with the proximity of theelectronic device to the physical object, wherein presenting the firstportion of the virtual environment at the first transparency includescombining the first portion of the virtual environment with firstimagery of the real environment; and presenting a second portion of thevirtual environment at a second transparency, different from the firsttransparency, associated with the proximity of the electronic device tothe physical object, wherein presenting the second portion of thevirtual environment at the second transparency includes combining thesecond portion of the virtual environment with second imagery of thereal environment; and in response to detecting that the proximity of theelectronic device to the physical object is greater than the firstthreshold distance: presenting the virtual environment without theisolated imagery of the physical object.
 20. The method of claim 19,further comprising: in response to detecting that the proximity of theelectronic device to the physical object is less than a second thresholddistance: modifying the first visibility value of the imagery of thephysical object.
 21. The method of claim 19, further comprising: inresponse to detecting that the proximity of the electronic device to thephysical object is less than a third threshold distance: ceasing topresent the virtual environment; and providing a view of the realenvironment.
 22. The method of claim 19, wherein the first portion ofthe virtual environment corresponds to the location of the physicalobject in the real environment, and wherein the second portion of thevirtual environment corresponds to a location different from thelocation of the physical object in the real environment.
 23. The methodof claim 19, wherein the first transparency and the second transparencyincrease as the proximity of the electronic device to the physicalobject decreases.